Hybrid air and liquid coolant conditioning unit for facilitating cooling of one or more electronics racks of a data center

ABSTRACT

A hybrid air and liquid coolant conditioning unit is provided for facilitating cooling of electronics rack(s) of a data center. The unit includes a first heat exchange assembly, including a liquid-to-liquid heat exchanger, a system coolant loop and a facility coolant loop, and a second heat exchange assembly, including an air-to-liquid heat exchanger, an air-moving device, and the facility coolant loop. The system coolant loop provides cooled system coolant to the electronics rack(s), and expels heat in the liquid-to-liquid heat exchanger from the electronics rack(s) to the facility coolant. The air-to-liquid heat exchanger extracts heat from the air of the data center and expels the heat to the facility coolant of the facility coolant loop. The facility coolant loop provides chilled facility coolant in parallel to the liquid-to-liquid heat exchanger and the air-to-liquid heat exchanger. In one implementation, the hybrid coolant conditioning unit includes a vapor-compression heat exchange assembly.

CROSS-REFERENCE TO RELATED PATENTS/APPLICATIONS

This application is a divisional application from U.S. patentapplication Ser. No. 11/944,680, filed Nov. 26, 2007, which published onMay 28, 2009, as U.S. Patent Publication No. 2009/0133866 A1, entitled“Hybrid Air and Liquid Coolant Conditioning Unit for FacilitatingCooling of One or More Electronics Racks of a Data Center”, the entiretyof which is hereby incorporated herein by reference.

TECHNICAL FIELD

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system level. Increased air flow rates are neededto effectively cool high power modules and to limit the temperature ofthe air that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable drawer configurations stacked within a rack orframe. In other cases, the electronics may be in fixed locations withinthe rack or frame. Typically, the components are cooled by air moving inparallel air flow paths, usually front-to-back, impelled by one or moreair moving devices (e.g., fans or blowers). In some cases it may bepossible to handle increased power dissipation within a single drawer byproviding greater air flow, through the use of a more powerful airmoving device or by increasing the rotational speed (i.e., RPMs) of anexisting air moving device. However, this approach is becomingproblematic at the rack level in the context of a computer installation(e.g., data center).

BRIEF SUMMARY

The sensible heat load carried by the air exiting the rack is stressingthe ability of the room air conditioning to effectively handle the load.This is especially true for large installations with “server farms” orlarge banks of computer racks close together. In such installations notonly will the room air conditioning be challenged, but the situation mayalso result in recirculation problems with some fraction of the “hot”air exiting one rack unit being drawn into the air inlet of the samerack or a nearby rack. This re-circulating flow is often extremelycomplex in nature, and can lead to significantly higher rack inlettemperatures than expected. In such installations, liquid cooling (e.g.,water cooling) is an attractive technology to assist in managing thehigher heat fluxes. The liquid absorbs the heat dissipated by thecomponents/modules in an efficient manner, and the heat can beultimately transferred from the liquid to an outside environment,whether air or other liquid coolant.

The shortcomings of the prior art are overcome and additional advantagesare thus provided through provision of an apparatus for facilitatingcooling of one or more electronics racks of a data center. The apparatusincludes a hybrid air and liquid coolant conditioning unit. The hybridair and liquid coolant conditioning unit includes a first heat exchangeassembly and a second heat exchange assembly. The first heat exchangeassembly includes a liquid-to-liquid heat exchanger, a system coolantloop and a facility coolant loop. When the hybrid air and liquid coolantconditioning unit is operational within the data center, the facilitycoolant loop receives chilled facility coolant from a source and passesat least a portion thereof through the liquid-to-liquid heat exchanger,and the system coolant loop provides cooled system coolant to the atleast one electronics rack, and expels heat via the liquid-to-liquidheat exchanger from the at least one electronics rack to the chilledfacility coolant in the facility coolant loop. The second heat exchangeassembly includes an air-to-liquid heat exchanger, an air-moving device(for moving air across the air-to-liquid heat exchanger), and thefacility coolant loop. When the hybrid air and liquid coolantconditioning unit is operational within the data center, the facilitycoolant loop receives chilled facility coolant from the source andpasses at least a portion thereof through the air-to-liquid heatexchanger. The air-to-liquid heat exchanger extracts heat from air ofthe data center moving across the air-to-liquid heat exchanger andexpels the heat to the chilled facility coolant in the facility coolantloop, thereby facilitating further cooling of the at least oneelectronics rack. The facility coolant loop provides chilled facilitycoolant in parallel to the liquid-to-liquid heat exchanger of the firstheat exchange assembly and to the air-to-liquid heat exchanger of thesecond heat exchange assembly. The hybrid air and liquid coolantconditioning unit further includes at least one control valve disposedwithin the facility coolant loop, wherein the facility coolant loopincludes a facility coolant supply line. The facility coolant supplyline feeds in parallel a first facility coolant supply inlet line to theliquid-to-liquid heat exchanger and a second facility coolant supplyinlet line to the air-to-liquid heat exchanger. The at least one controlvalve is coupled to at least one of the first facility coolant supplyinlet line or the second facility coolant supply inlet line forseparately controlling facility coolant flow through at least one of theliquid-to-liquid heat exchanger or the air-to-liquid heat exchanger.

In another aspect, a data center is provided herein which includes atleast one electronics rack and a hybrid air and liquid coolantconditioning unit. Each electronics rack of the at least one electronicsrack has an air inlet side and an air outlet side, the air inlet andoutlet sides respectively enabling ingress and egress of air through theelectronics rack. The hybrid air and liquid coolant conditioning unitincludes a first heat exchange assembly and a second heat exchangeassembly. The first heat exchange assembly includes a liquid-to-liquidheat exchanger, a system coolant loop and a facility coolant loop, andthe second heat exchange assembly includes an air-to-liquid heatexchanger, an air-moving device and the facility coolant loop, whereinthe facility coolant loop is shared between the first heat exchangeassembly and the second heat exchange assembly. When the hybrid air andliquid coolant conditioning unit is operational within the data center,the facility coolant loop receives chilled facility coolant from asource and passes at least a portion thereof through theliquid-to-liquid heat exchanger, and the system coolant loop providescooled system coolant to the at least one electronics rack, and expelsheat via the liquid-to-liquid heat exchanger from the at least oneelectronics rack to the chilled facility coolant in the facility coolantloop. Further, when the hybrid air and liquid coolant conditioning unitis operational, the air-moving device moves air across the air-to-liquidheat exchanger, and at least a portion of facility coolant from thesource passes through the air-to-liquid heat exchanger, where theair-to-liquid heat exchanger extracts heat from air of the data centermoving across the air-to-liquid heat exchanger and expels the heat tothe chilled facility coolant in the facility coolant loop, therebyfurther facilitating cooling of the at least one electronics rack. Theshared facility coolant loop provides chilled facility coolant inparallel to the liquid-to-liquid heat exchanger of the first heatexchange assembly and to the air-to-liquid heat exchanger of the secondheat exchange assembly. The hybrid air and liquid coolant conditioningunit further includes at least one control valve disposed within thefacility coolant loop, wherein the facility coolant loop includes afacility coolant supply line. The facility coolant supply line feeds inparallel a first facility coolant supply inlet line to theliquid-to-liquid heat exchanger and a second facility coolant supplyinlet line to the air-to-liquid heat exchanger. The at least one controlvalve is coupled to at least one of the first facility coolant supplyinlet line or the second facility coolant supply inlet line forseparately controlling facility coolant flow through at least one of theliquid-to-liquid heat exchanger or the air-to-liquid heat exchanger.

In a further aspect, a method of cooling at least one electronics rackof a data center is provided. The method includes: obtaining a hybridair and liquid coolant conditioning unit, the hybrid air and liquidcoolant conditioning unit including a first heat exchange assembly and asecond heat exchange assembly, the first heat exchange assemblycomprising a liquid-to-liquid heat exchanger, a system coolant loop, andat least a portion of a facility coolant loop, the second heat exchangeassembly comprising an air-to-liquid heat exchanger, an air-movingdevice, and at least a portion of the facility coolant loop; disposingthe hybrid air and liquid coolant conditioning unit in operativeposition with a data center comprising at least one electronics rack;operating the hybrid air and liquid cooling conditioning unit to coolthe at least one electronics rack of the data center, the operatingincluding: employing the facility coolant loop to pass at least aportion of chilled facility coolant from a source through theliquid-to-liquid heat exchanger, and controlling the portion passingthrough the liquid-to-liquid heat exchanger employing at least onecontrol valve of the hybrid air and liquid coolant conditioning unit;providing cooled system coolant to the at least one electronics rackfrom the liquid-to-liquid heat exchanger, and expelling heat via theliquid-to-liquid heat exchanger from the at least one electronics rackto the chilled facility coolant in the facility coolant loop, whereinthe at least one control valve allows adjustment of facility coolantflow through the liquid-to-liquid heat exchanger, thereby allowingcontrol of temperature of system coolant in the system coolant loop forfacilitating cooling of at least one electronics rack; and operating theair-moving device to move air across the air-to-liquid heat exchangerwhile passing at least a portion of chilled facility coolant from thefacility coolant loop through the air-to-liquid heat exchanger, whereinthe air-to-liquid heat exchanger extracts heat from data center airmoving across the air-to-liquid heat exchanger and expels the heat tothe chilled facility coolant in the facility coolant loop, therebyfurther cooling the at least one electronics rack of the data center,wherein the facility coolant loop provides chilled facility coolant inparallel to the liquid-to-liquid heat exchanger of the first heatexchange assembly and to the air-to-liquid heat exchanger of the secondheat exchange assembly.

Further, additional features and advantages are realized through thetechniques of the present invention. Other embodiments and aspects ofthe invention are described in detail herein and are considered a partof the claimed invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2 depicts recirculation airflow patterns to be addressed in oneimplementation of a raised floor layout of an air-cooled data center, inaccordance with an aspect of the present invention;

FIG. 3 depicts one embodiment of a coolant distribution unit for liquidcooling of one or more electronics racks of a data center, in accordancewith an aspect of the present invention;

FIG. 4 is a plan view of one embodiment of an electronics subsystemlayout illustrating an air and liquid cooling subsystem for hybridcooling of components of the electronics subsystem, in accordance withan aspect of the present invention;

FIG. 5 depicts one detailed embodiment of a partially-assembledelectronics subsystem layout, wherein the electronics subsystem includeseight heat-generating electronics components to be actively cooled, eachhaving a respective liquid-cooled cold plate of a liquid-based coolingsystem coupled thereto, in accordance with an aspect of the presentinvention;

FIG. 6 is a plan view of one embodiment of a data center layoutemploying a hybrid air and liquid coolant conditioning unit, inaccordance with an aspect of the present invention;

FIG. 7 is a schematic of one embodiment of a hybrid air and liquidcoolant conditioning unit, in accordance with an aspect of the presentinvention;

FIG. 8 is a plan view of one embodiment of a data center layoutemploying an alternate hybrid air and liquid coolant conditioning unit,in accordance with an aspect of the present invention; and

FIG. 9 is a schematic of one embodiment of the alternate hybrid air andliquid coolant conditioning unit of FIG. 8, in accordance with an aspectof the present invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronics system, and may be, for example, a standalone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise multipleelectronics subsystems, each having one or more heat generatingcomponents disposed therein requiring cooling. “Electronics subsystem”refers to any sub-housing, blade, book, drawer, node, compartment, etc.,having one or more heat generating electronic components disposedtherein. Each electronics subsystem of an electronics rack may bemovable or fixed relative to the electronics rack, with the rack-mountedelectronics drawers of a multi-drawer rack unit and blades of a bladecenter system being two examples of subsystems of an electronics rack tobe cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. As used herein, “primary heat generating component”refers to a primary heat generating electronic component within anelectronics subsystem, while “secondary heat generating component”refers to an electronic component of the electronics subsystemgenerating less heat than the primary heat generating component to becooled. “Primary heat generating die” refers, for example, to a primaryheat generating die or chip within a heat generating electroniccomponent comprising primary and secondary heat generating dies (with aprocessor die being one example). “Secondary heat generating die” refersto a die of a multi-die electronic component generating less heat thanthe primary heat generating die thereof (with memory dies and memorysupport dies being examples of secondary dies to be cooled). As oneexample, a heat generating electronic component could comprise multipleprimary heat generating bare dies and multiple secondary heat generatingdies on a common carrier. Further, unless otherwise specified herein,the term “liquid-cooled cold plate” refers to any conventional thermallyconductive structure having a plurality of channels or passagewaysformed therein for flowing of liquid coolant therethrough. In addition,“metallurgically bonded” refers generally herein to two components beingwelded, brazed or soldered together by any means.

As used herein, “air-to-liquid heat exchanger” means any heat exchangemechanism characterized as described herein through which liquid coolantcan circulate and which transfers heat between air and the circulatingliquid; and includes, one or more discrete air-to-liquid heat exchangerscoupled either in series or in parallel. An air-to-liquid heat exchangermay comprise, for example, one or more coolant flow paths, formed ofthermally conductive tubing (such as copper or other tubing) in thermalcommunication with a plurality of air-cooled cooling fins. Size,configuration and construction of the air-to-liquid heat exchanger canvary without departing from the scope of the invention disclosed herein.A “liquid-to-liquid heat exchanger” may comprise, for example, two ormore coolant flow paths, formed of thermally conductive tubing (such ascopper or other tubing) in thermal communication with each other. Size,configuration and construction of the liquid-to-liquid heat exchangercan vary without departing from the scope of the invention disclosedherein. Further, “data center” refers to a computer installationcontaining one or more electronics racks to be cooled. As a specificexample, a data center may include one or more rows of rack-mountedcomputing units, such as server units.

One example of facility coolant and system coolant is water. However,the concepts disclosed herein are readily adapted to use with othertypes of coolant on the facility side and/or on the system side. Forexample, one or more of the coolants may comprise a brine, afluorocarbon liquid, a liquid metal, or other similar coolant, orrefrigerant. In another example described herein, the facility coolantis a refrigerant, while the system coolant is water. All of thesevariations are possible, while still maintaining the advantages andunique features of the present invention.

Reference is made below to the drawings, which are not drawn to scalefor reasons of understanding, wherein the same reference numbers usedthroughout different figures designate the same or similar components.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic components within thedrawer(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more air conditioning units 150, also disposed within the datacenter 100. Room air is taken into each air conditioning unit 150 nearan upper portion thereof. This room air comprises in part exhausted airfrom the “hot” aisles of the computer installation defined by opposingair outlet sides 130 of the electronics racks 110.

Due to the ever increasing airflow requirements through electronicsracks, and limits of air distribution within the typical data centerinstallation, recirculation problems within the room may occur. This isshown in FIG. 2 for a raised floor layout, wherein hot air recirculation200 occurs from the air outlet sides 130 of the electronics racks 110back to the cold air aisle defined by the opposing air inlet sides 120of the electronics rack. This recirculation can occur because theconditioned air supplied through tiles 160 is typically only a fractionof the airflow rate forced through the electronics racks by the airmoving devices disposed therein. This can be due, for example, tolimitations on the tile sizes (or diffuser flow rates). The remainingfraction of the supply of inlet side air is often made up by ambientroom air through recirculation 200. This recirculating flow is oftenvery complex in nature, and can lead to significantly higher rack unitinlet temperatures than desired.

The recirculation of hot exhaust air from the hot aisle of the computerroom installation to the cold aisle can be detrimental to theperformance and reliability of the computer system(s) or electronicsystem(s) within the racks. Data center equipment is typically designedto operate with rack air inlet temperatures in the 18-35° C. range. Fora raised floor layout such as depicted in FIG. 1, however, temperaturescan range from 15-20° C. at the lower portion of the rack, close to thecooled air input floor vents, to as much as 45-50° C. at the upperportion of the electronics rack, where the hot air can form aself-sustaining recirculation loop. Since the allowable rack heat loadis limited by the rack inlet air temperature at the “hot” part, thistemperature distribution correlates to an inefficient utilization ofavailable chilled air. Also, computer installation equipment almostalways represents a high capital investment to the customer. Thus, it isof significant importance, from a product reliability and performanceview point, and from a customer satisfaction and business perspective,to limit the temperature of the inlet air to the rack unit to besubstantially uniform. The efficient cooling of such computer andelectronic systems, and the amelioration of localized hot air inlettemperatures to one or more rack units due to recirculation of aircurrents, are addressed by the apparatuses and methods disclosed herein,as is reducing acoustic noise within the data center (e.g., by requiringless cooled air within the data center and less cooled airflow throughthe electronics racks to remove a given heat load, thereby reducingair-moving device requirements and hence acoustic noise within the datacenter).

FIG. 3 depicts one embodiment of a coolant distribution unit 300 for adata center. The coolant distribution unit is conventionally a largeunit which occupies what would be considered a full electronics frame.Within coolant distribution unit 300 is a power/control element 312, areservoir/expansion tank 313, a heat exchanger 314, a pump 315 (oftenaccompanied by a redundant second pump), facility water inlet 316 andoutlet 317 supply pipes, a supply manifold 318 supplying water or systemcoolant to the electronics racks 110 via couplings 320 and lines 322,and a return manifold 319 receiving water from the electronics racks110, via lines 323 and couplings 321. Each electronics rack includes (inone example) a power/control unit 330 for the electronics rack, multipleelectronics subsystems 340, a system coolant supply manifold 350, and asystem coolant return manifold 360. As shown, each electronics rack 110is disposed on raised floor 140 of the data center and lines 323providing system coolant to system coolant supply manifolds 350 andlines 322 facilitating return of system coolant from system coolantreturn manifolds 360 are disposed in the supply air plenum beneath theraised floor.

In the embodiment illustrated, the system coolant supply manifold 350provides system coolant to the cooling systems of the electronicssubsystems (more particularly, to liquid-cooled cold plates thereof) viaflexible hose connections 351, which are disposed between the supplymanifold and the respective electronics subsystems within the rack.Similarly, system coolant return manifold 360 is coupled to theelectronics subsystems via flexible hose connections 361. Quick connectcouplings may be employed at the interface between flexible hoses 351,361 and the individual electronics subsystems. By way of example, thesequick connect couplings may comprise various types of commerciallyavailable couplings, such as those available from Colder ProductsCompany, of St. Paul, Minn., USA, or Parker Hannifin, of Cleveland,Ohio, USA.

Although not shown, electronics rack 110 may also include anair-to-liquid heat exchanger disposed at an air outlet side thereof,which also receives system coolant from the system coolant supplymanifold 350 and returns system coolant to the system coolant returnmanifold 360.

FIG. 4 depicts one embodiment of an electronics subsystem 340 componentlayout wherein one or more air moving devices 411 provide forced airflow 415 to cool multiple components 412 within electronics subsystem340. Cool air is taken in through a front 431 and exhausted out a back433 of the drawer. The multiple components to be cooled include multipleprocessor modules to which liquid-cooled cold plates 420 (of aliquid-based cooling system) are coupled, as well as multiple arrays ofmemory modules 430 (e.g., dual in-line memory modules (DIMMs)) andmultiple rows of memory support modules 432 (e.g., DIMM control modules)to which air-cooled heat sinks are coupled. In the embodimentillustrated, memory modules 430 and the memory support modules 432 arepartially arrayed near front 431 of electronics subsystem 340, andpartially arrayed near back 433 of electronics subsystem 340. Also, inthe embodiment of FIG. 4, memory modules 430 and the memory supportmodules 432 are cooled by air flow 415 across the electronics subsystem.

The illustrated liquid-based cooling system further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 420. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 440, a bridge tube 441 and a coolant return tube442. In this example, each set of tubes provides liquid coolant to aseries-connected pair of cold plates 420 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 440 and from the first cold plate to a second coldplate of the pair via bridge tube or line 441, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 442.

FIG. 5 depicts in greater detail an alternate electronics drawer layoutcomprising eight processor modules, each having a respectiveliquid-cooled cold plate of a liquid-based cooling system coupledthereto. The liquid-based cooling system is shown to further includeassociated coolant-carrying tubes for facilitating passage of liquidcoolant through the liquid-cooled cold plates and a header subassemblyto facilitate distribution of liquid coolant to and return of liquidcoolant from the liquid-cooled cold plates. By way of specific example,the liquid coolant passing through the liquid-based cooling subsystem ischilled water.

As noted, various liquid coolants significantly outperform air in thetask of removing heat from heat generating electronic components of anelectronics system, and thereby more effectively maintain the componentsat a desirable temperature for enhanced reliability and peakperformance. As liquid-based cooling systems are designed and deployed,it is advantageous to architect systems which maximize reliability andminimize the potential for leaks while meeting all other mechanical,electrical and chemical requirements of a given electronics systemimplementation. These more robust cooling systems have unique problemsin their assembly and implementation. For example, one assembly solutionis to utilize multiple fittings within the electronics system, and useflexible plastic or rubber tubing to connect headers, cold plates, pumpsand other components. However, such a solution may not meet a givencustomer's specifications and need for reliability.

Thus, presented herein in one aspect is a robust and reliableliquid-based cooling system specially preconfigured and prefabricated asa monolithic structure for positioning within a particular electronicsdrawer.

FIG. 5 is an isometric view of one embodiment of an electronics drawerand monolithic cooling system, in accordance with an aspect of thepresent invention. The depicted planar server assembly includes amulti-layer printed circuit board to which memory DIMM sockets andvarious electronic components to be cooled are attached both physicallyand electrically. In the cooling system depicted, a supply header isprovided to distribute liquid coolant from a single inlet to multipleparallel coolant flow paths and a return header collects exhaustedcoolant from the multiple parallel coolant flow paths into a singleoutlet. Each parallel coolant flow path includes one or more cold platesin series flow arrangement to cool one or more electronic components towhich the cold plates are mechanically and thermally coupled. The numberof parallel paths and the number of series-connected liquid-cooled coldplates depends, for example on the desired device temperature, availablecoolant temperature and coolant flow rate, and the total heat load beingdissipated from each electronic component.

More particularly, FIG. 5 depicts a partially assembled electronicssystem 513 and an assembled liquid-based cooling system 515 coupled toprimary heat generating components (e.g., including processor dies) tobe cooled. In this embodiment, the electronics system is configured for(or as) an electronics drawer of an electronics rack, and includes, byway of example, a support substrate or planar board 505, a plurality ofmemory module sockets 510 (with the memory modules (e.g., dual in-linememory modules) not shown), multiple rows of memory support modules 532(each having coupled thereto an air-cooled heat sink 534), and multipleprocessor modules (not shown) disposed below the liquid-cooled coldplates 520 of the liquid-based cooling system 515.

In addition to liquid-cooled cold plates 520, liquid-based coolingsystem 515 includes multiple coolant-carrying tubes, including coolantsupply tubes 540 and coolant return tubes 542 in fluid communicationwith respective liquid-cooled cold plates 520. The coolant-carryingtubes 540, 542 are also connected to a header (or manifold) subassembly550 which facilitates distribution of liquid coolant to the coolantsupply tubes 540 and return of liquid coolant from the coolant returntubes 542. In this embodiment, the air-cooled heat sinks 534 coupled tomemory support modules 532 closer to front 531 of electronics drawer 513are shorter in height than the air-cooled heat sinks 534′ coupled tomemory support modules 532 near back 533 of electronics drawer 513. Thissize difference is to accommodate the coolant-carrying tubes 540, 542since, in this embodiment, the header subassembly 550 is at the front531 of the electronics drawer and the multiple liquid-cooled cold plates520 are in the middle of the drawer.

Liquid-based cooling system 515 comprises a preconfigured monolithicstructure which includes multiple (pre-assembled) liquid-cooled coldplates 520 configured and disposed in spaced relation to engagerespective heat generating electronic components. Each liquid-cooledcold plate 520 includes, in this embodiment, a liquid coolant inlet anda liquid coolant outlet, as well as an attachment subassembly (i.e., acold plate/load arm assembly). Each attachment subassembly is employedto couple its respective liquid-cooled cold plate 520 to the associatedelectronic component to form the cold plate and electronic componentassemblies. Alignment openings (i.e., thru-holes) are provided on thesides of the cold plate to receive alignment pins or positioning dowelsduring the assembly process, as described further in theabove-incorporated patent application entitled “Method of Assembling aCooling System for a Multi-Component Electronics System”. Additionally,connectors (or guide pins) are included within attachment subassemblywhich facilitate use of the attachment assembly.

As shown in FIG. 5, header subassembly 550 includes two liquidmanifolds, i.e., a coolant supply header 552 and a coolant return header554, which in one embodiment, are coupled together via supportingbrackets. In the monolithic cooling structure of FIG. 5, the coolantsupply header 552 is metallurgically bonded in fluid communication toeach coolant supply tube 540, while the coolant return header 554 ismetallurgically bonded in fluid communication to each coolant returntube 552. A single coolant inlet 551 and a single coolant outlet 553extend from the header subassembly for coupling to the electronicsrack's coolant supply and return manifolds (not shown).

FIG. 5 also depicts one embodiment of the preconfigured,coolant-carrying tubes. In addition to coolant supply tubes 540 andcoolant return tubes 542, bridge tubes or lines 541 are provided forcoupling, for example, a liquid coolant outlet of one liquid-cooled coldplate to the liquid coolant inlet of another liquid-cooled cold plate toconnect in series fluid flow the cold plates, with the pair of coldplates receiving and returning liquid coolant via a respective set ofcoolant supply and return tubes. In one embodiment, the coolant supplytubes 540, bridge tubes 541 and coolant return tubes 542 are eachpreconfigured, semi-rigid tubes formed of a thermally conductivematerial, such as copper or aluminum, and the tubes are respectivelybrazed, soldered or welded in a fluid-tight manner to the headersubassembly and/or the liquid-cooled cold plates. The tubes arepreconfigured for a particular electronics system to facilitateinstallation of the monolithic structure in engaging relation with theelectronics system.

The hybrid air and liquid coolant conditioning unit configurationsdepicted hereinbelow with reference to FIGS. 6-9 may be employed withair and liquid cooling subsystems such as described above in connectionwith FIGS. 4 & 5. Alternatively, the hybrid air and liquid coolantconditioning unit could be employed to either air-cool only orliquid-cool only the electronics racks, dependent on the data centerimplementation.

FIG. 6 depicts one embodiment of a data center 600 layout whereinmultiple rows of electronics racks 110 are cooled employing a hybrid airand liquid coolant conditioning unit 610, in accordance with an aspectof the present invention. As shown, hybrid air and liquid coolantconditioning unit 610 receives (via a facility coolant supply line 601)chilled facility coolant and returns facility coolant (via a facilitycoolant return line 602) to a source. Cooled air 611 is output from thehybrid air and liquid coolant conditioning unit, for example, into acold air plenum of a raised floor data center embodiment, such asdepicted in FIG. 1. The cooled air 611 is produced by the hybrid air andliquid coolant conditioning unit extracting heat from warm data centerair 612 being drawn through the hybrid air and liquid coolantconditioning unit, for example, by one or more air moving devices. Thiswarm data center air 612 comprises, in part, air exhausted from the airoutlet sides of the electronics racks within the data center. Hybrid airand liquid coolant conditioning unit 610 also outputs chilled systemcoolant (via a system coolant supply line 621) and receives systemcoolant from the electronics racks (via a system coolant return line622). Together, system coolant supply line 621 and system coolant returnline 622, and the structures in fluid communication therewith, define asystem coolant loop between the electronics racks within the data centerand the hybrid air and liquid coolant conditioning unit.

FIG. 7 depicts one embodiment of hybrid air and liquid coolantconditioning unit 610. In this embodiment, the facility coolant isassumed to be, for example, chilled water received via facility coolantsupply line 601 and returned via facility coolant return line 602.Together, these facility coolant lines, and the structures in fluidcommunication therewith, form a facility coolant loop 750.

As shown in FIG. 7, within hybrid air and liquid coolant conditioningunit 610 are a first heat exchange assembly 710 and a second heatexchange assembly 720. First heat exchange assembly 710 includes asystem coolant loop 740 through which system coolant circulates betweenthe first heat exchange assembly and the one or more electronics racksof the data center to be liquid-cooled. Additionally, first heatexchange assembly 710 includes a liquid-to-liquid heat exchanger 712 anda portion of facility coolant loop 750. When operational within the datacenter, facility coolant loop 750 receives chilled facility coolant froma source and passes at least a portion thereof through liquid-to-liquidheat exchanger 712, the portion being controlled by at least one firstcontrol valve 713 disposed (in this implementation) within a firstfacility coolant supply inlet line 703 to liquid-to-liquid heatexchanger 712. Similarly, a first facility coolant return outlet line707 couples liquid-to-liquid heat exchanger 712 to facility coolantreturn line 602 of facility coolant loop 750.

System coolant loop 740 provides cooled system coolant to one or moreelectronics racks via a supply manifold 715 and one or more systemcoolant supply lines 621 coupling the hybrid air and liquid coolantconditioning unit to one or more electronics racks of the data center.Exhausted system coolant is returned via one or more system coolantreturn lines 622 from the one or more electronics racks through a returnmanifold 716 and a coolant pump 711 to liquid-to-liquid heat exchanger712 where heat is expelled to the chilled facility coolant in thefacility coolant loop 750. A first temperature sensor 714 is coupled tosystem coolant loop 740 to monitor temperature of system coolant withinthe loop, for example, at an outlet of the liquid-to-liquid heatexchanger 712, as illustrated. Sensed temperature values are provided toa control unit 725, which includes logic to automatically adjustfacility coolant flow through liquid-to-liquid heat exchanger 712employing the at least one first control valve 713. By increasing ordecreasing facility coolant flow through liquid-to-liquid heat exchanger712, temperature of system coolant in the system coolant loop can beadjusted, for example, to remain at a desired system coolant set point,or within a desired system coolant temperature range, such as above aroom dew point temperature. One skilled in the art can readily implementlogic to accomplish this automatic adjustment function. For example, theat least one first control valve 713 may be step-wise opened or closed,dependent on sensed temperature at the outlet of liquid-to-liquid heatexchanger 712 to either increase or decrease, respectively, facilitycoolant flow through the liquid-to-liquid heat exchanger.

Second heat exchange assembly 720 includes an air-moving device 721, anair-to-liquid heat exchanger 722 and a portion of facility coolant loop750. When operational, air-moving device 721 moves warm data center air612 across air-to-liquid heat exchanger 722, which extracts heat fromthe air and expels the heat to the chilled facility coolant in facilitycoolant loop 750. This heat originates, at least partially, with one ormore electronics racks disposed within the data center room beingair-cooled. Air-to-liquid heat exchanger 722 exhausts cooled air 611,for example, to a cold air plenum in a raised floor data centerembodiment. Alternatively, the cooled air 611 could be exhausteddirectly into the data center to facilitate maintaining temperature ofthe data center at a desired temperature.

Second heat exchange assembly 720 also includes at least one secondcontrol valve 723 coupled to a second facility coolant supply inlet line705, which is in fluid communication with facility coolant supply line601. As shown, facility coolant supply line 601 provides facilitycoolant in parallel to liquid-to-liquid heat exchanger 712 of first heatexchange assembly 710 and to air-to-liquid heat exchanger 722 of secondheat exchange assembly 720 via first facility coolant supply inlet line703 and second facility coolant supply inlet line 705, respectively.Similarly, second heat exchange assembly 712 includes a second facilitycoolant return outlet line 709 in fluid communication with facilitycoolant return line 602 for exhausting facility coolant for return, forexample, to the facility coolant source (not shown). Second heatexchange assembly 720 also includes a second temperature sensor 724 forsensing air temperature within the data center. For example, secondtemperature sensor 724 could be disposed in the raised floor cold airplenum in the embodiment of FIG. 1. As with the first heat exchangeassembly, control unit 725 is coupled to the second temperature sensor724 and to the at least one second control valve 723 for automaticallyadjusting facility coolant flow through the air-to-liquid heatexchanger, thereby controlling air temperature expelled from the secondheat exchange assembly to facilitate cooling of the electronics rackwithin the data center. One skilled in the art can again readilyimplement logic within control unit 725 to accomplish this temperaturemonitoring and control valve adjusting function. In one implementation,control valves 713, 723 are motor driven control valves.

In the embodiment illustrated in FIG. 7, a motor 730 simultaneouslydrives pump 711 of first heat exchange assembly 710 and air-movingdevice 721 of second heat exchange assembly 720. This may beaccomplished, for example, by employing a double-shafted motor, such asavailable from AmericanHVACParts.com of Rancho Cucamonga, Calif., USA.In another implementation, an off the shelf transmission could beemployed which accepts a single rotating shaft torque input and througha set of internal gears outputs one, two, or several rotating torqueoutputs at whatever desired speeds are requested. Another approach is touse a motor equipped with a pulley or sprocket, and equip each pump andair-moving device with a receiving pulley or sprocket (sized to achievethe appropriate shaft speed for the pump or air-moving device). Themotor and pump(s) and air-moving device(s) would then be connected by abelt or chain (among the pulleys or sprockets, respectively).

FIG. 8 depicts an alternate hybrid air and liquid coolant conditioningunit 610′ for data center 600, which again includes one or more rows ofelectronics racks 110 to be air and/or liquid-cooled. Hybrid air andliquid coolant conditioning unit 610′ again provides cooled air 611 forair-cooling of the one or more electronics racks, and provides systemcoolant via a system coolant loop (comprising system coolant supply line621 and system coolant return line 622) to the one or more electronicsracks. Electronics racks 110 employ air-moving devices, for example,within the respective racks, to draw cool air from a cold air aislewithin the data center through the one or more electronics subsystems ofthe rack for exhausting out an air outlet side thereof as warm air 612,which is then drawn back to the hybrid and air and liquid coolantconditioning unit, for example, via one or more air-moving devicesdisposed within the hybrid air and liquid coolant conditioning unit.

In this embodiment, the hybrid air and liquid coolant conditioning unit610′ comprises a vapor-compression heat exchange assembly which, asshown, includes a vapor-compression unit 800 disposed outside datacenter 600. Vapor-compression unit 800 includes a compressor and acondenser disposed in fluid communication with a facility coolant loopcomprising a facility coolant supply line 801 and a facility coolantreturn line 802. In this embodiment, the facility coolant is arefrigerant flowing through the facility coolant loop. The condenser ofvapor-compression unit 800 is one of an air-cooled condenser unit (e.g.,employing outdoor air 810 for cooling) or a liquid-cooled condenser unit(e.g., employing chilled water (not shown)).

FIG. 9 depicts one implementation of the hybrid air and liquid coolantconditioning unit 610′ of FIG. 8, wherein the liquid-to-liquid heatexchanger 712′ functions as a system coolant conditioning evaporator,and the air-to-liquid heat exchanger 722′ functions as anair-conditioning evaporator. As shown in FIG. 9, the facility coolantloop 902, comprising facility coolant supply line 801 and facilitycoolant return line 802, includes a first facility coolant supply inletline 903 and a second facility coolant supply inlet line 905 couplingfacility coolant supply line 801 to the respective heat exchangers. Eachfacility coolant supply inlet line 903, 905 includes a respectiveexpansion valve 910, 911 through which facility coolant passes.

In operation, the vapor-compression heat exchange system uses acirculating refrigerant as the medium which absorbs and removes heatfrom air and system coolant egressing from one or more electronics racksof the data center, and subsequently rejects the heat via the condenserof the vapor-compression unit. Circulating refrigerant enters thecompressor (not shown) of the vapor-compression unit (see FIG. 8) in thethermodynamic state known as a saturated or superheated vapor and iscompressed to a higher pressure, resulting in a higher temperature aswell. The hot, compressed vapor is then in the thermodynamic state knownas a super-heated vapor and it is at a temperature and pressure at whichit can be condensed with typically available cooling water or coolingair. The hot vapor is routed through a condenser (not shown) of thevapor-compression unit (see FIG. 8) where it is cooled and condensedinto a liquid by flowing through a coil or tubes with cool water or coolair flowing across the coil or tubes. This is where the circulatingrefrigerant rejects heat from the system and the rejected heat iscarried away by either the water or the air, depending on thevapor-compression unit implementation.

The condensed liquid refrigerant, in the thermodynamic state known as asaturated liquid, is next routed through one of the expansion valves910, 911, which as noted above, are disposed in the facility coolantsupply inlet lines 903, 905, respectively. Passing through the expansionvalve, the refrigerant undergoes an abrupt reduction in pressure. Thispressure reduction results in the flash evaporation of a part of theliquid refrigerant. The auto-refrigeration effect of flash evaporationfurther lowers the temperature of the liquid and vapor-refrigerantmixture. The cold mixture is then routed through an evaporator (i.e.,the liquid-to-liquid heat exchanger and/or the air-to-liquid heatexchanger). Warm system coolant is pumped via pump 711 in the first heatexchange assembly 710′ through liquid-to-liquid heat exchanger 712′, andair-moving device 721 of second heat exchange assembly 720′ establishesa warm airflow across air-to-liquid heat exchanger 722′. The warm systemcoolant and the warm airflow pass through the respective heatexchangers, with cold system coolant being output in the system coolantloop 901 from liquid-to-liquid heat exchanger 712′, and cooled air 611egressing from the air-to-liquid heat exchanger 722′. The warm systemcoolant and warm air passing through the respective heat exchangersevaporates the liquid part of the cold refrigerant mixture within theevaporator coil (i.e., the heat exchanger). At the same time, the systemcoolant is cooled, and the circulating air is cooled, thus lowering thetemperature of the system coolant and the air output from the hybrid airand liquid coolant conditioning unit, respectively. The heat exchangers(i.e., evaporator coils), are thus where the circulating refrigerantabsorbs and removes heat, which is subsequently rejected in thecondenser and transferred elsewhere by the water or air used to cool thecondenser. The refrigeration cycle then repeats, with the refrigerantvapor from the evaporators being routed back to the compressor of thevapor-compression unit.

As shown in FIG. 9, first heat exchange assembly 710′ further includesat least one first control valve 920, disposed in communication withfacility coolant return outlet line 907 coupled to liquid-to-liquid heatexchanger 712′, and a first temperature sensor 714, disposed incommunication with system coolant loop 901 for sensing system coolanttemperature, for example, at an outlet of the liquid-to-liquid heatexchanger 712′, as illustrated. A control unit 925 is coupled to the atleast one first control valve 920 and first temperature sensor 714 foradjusting flow of facility coolant through the at least one firstcontrol valve 920, dependent, for example, on sensed temperature ofsystem coolant within system coolant loop 901. By locating the at leastone first control valve 920 downstream from liquid-to-liquid heatexchanger 712′, the control valve may be employed to control saturationpressure of refrigerant within the heat exchanger. As pressure ofrefrigerant within the heat exchanger (or more particularly, theevaporator), is adjusted, saturation temperature of the refrigerant isalso adjusted. If desired, expansion valves 910, 911 may be alsoadjustable, in which case control unit 925 is also coupled to expansionvalve 910 for directly controlling, for example, an orifice size of theexpansion valve. Similarly, control unit 925 is coupled to at least onesecond control valve 921 and to a second temperature sensor 724 ofsecond heat exchange assembly 720′. As illustrated, the at least onesecond control valve 921 is located in a second facility coolant returnoutlet line 909 coupling air-to-liquid heat exchanger 722′ to thefacility coolant return line 802 of the facility coolant loop 902. Thisagain allows control unit 925 to control saturation pressure ofrefrigerant within the air-to-liquid heat exchanger 722′ by controllingthe at least one second control valve 921 dependent, for example, on asensed air temperature obtained from second temperature sensor 724.Additionally, expansion valve 911 may be controllable, in which case thevalve would be coupled to control unit 925, for example, for automaticadjustment of an orifice size within the expansion valve.

As in the embodiment of FIG. 7, a single motor 730 may againsimultaneously drive pump 711 of first heat exchange assembly 710′ andair-moving device 721 of second heat exchange assembly 720′. In analternate embodiment, separate motors may be employed to drive pump 711of first heat exchange assembly 710′ and air-moving device 721 of secondheat exchange assembly 720′.

Although preferred embodiments have been depicted and described indetail herein, it will be apparent to those skilled in the relevant artthat various modifications, additions, substitutions and the like can bemade without departing from the spirit of the invention and these aretherefore considered to be within the scope of the invention as definedin the following claims.

What is claimed is:
 1. A method of cooling multiple electronics racks ofa data center, the method comprising: obtaining a hybrid air and liquidcoolant conditioning unit, the hybrid air and liquid coolantconditioning unit including a first heat exchange assembly and a secondheat exchange assembly, the first heat exchange assembly comprising aliquid-to-liquid heat exchanger, a system coolant loop to facilitatefeeding system coolant cooled by the hybrid air and liquid coolantconditioning unit to multiple electronics racks for cooling thereof, anda first portion of a facility coolant loop through which chilledfacility coolant is to flow, the second heat exchange assemblycomprising an air-to-liquid heat exchanger, an air-moving device, and asecond portion of the facility coolant loop; disposing the hybrid airand liquid coolant conditioning unit in operative position within a datacenter comprising the multiple electronics racks; operating the hybridair and liquid coolant conditioning unit to cool the the multipleelectronics racks of the data center, the operating including: employingthe facility coolant loop to pass at least a portion of the chilledfacility coolant from a source through the liquid-to-liquid heatexchanger of the first heat exchange assembly, and controlling theportion passed through the liquid-to-liquid heat exchanger employing atleast one control valve of the hybrid air and liquid coolantconditioning unit to control cooling of the system coolant in the systemcoolant loop; providing the system coolant to the multiple electronicsracks from the liquid-to-liquid heat exchanger of the first heatexchange assembly after cooling thereof within the liquid-to-liquid heatexchanger, and expelling heat via the liquid-to-liquid heat exchangerfrom the multiple electronics racks to the chilled facility coolant inthe facility coolant loop, wherein the at least one control valve allowsadjustment of facility coolant flow through the liquid-to-liquid heatexchanger, thereby allowing control of temperature of system coolant inthe system coolant loop for facilitating cooling of the multipleelectronics racks; operating the air-moving device to move air acrossthe air-to-liquid heat exchanger while passing the second portion ofchilled facility coolant from the facility coolant loop through theair-to-liquid heat exchanger of the second heat exchange assembly tocool data center air, wherein the air-to-liquid heat exchanger extractsheat from the data center air moving across the air-to-liquid heatexchanger and expels the heat to the chilled facility coolant in thefacility coolant loop, thereby further cooling the multiple electronicsracks of the data center, wherein the facility coolant loop provides thechilled facility coolant in parallel to the liquid-to-liquid heatexchanger of the first heat exchange assembly and to the air-to-liquidheat exchanger of the second heat exchange assembly.
 2. The method ofclaim 1, wherein the hybrid air and liquid coolant conditioning unitfurther comprises a first temperature sensor coupled to the systemcoolant loop for sensing temperature of system coolant therein, and asecond temperature sensor disposed to sense an air temperature withinthe data center, and a control unit coupled to the at least one controlvalve, the first temperature sensor, and the second temperature sensor,and wherein the method further comprises automatically adjusting atleast one of facility coolant flow through the liquid-to-liquid heatexchanger dependent on sensed temperature of system coolant within thesystem coolant loop ascertained via the first temperature sensor, orautomatically adjusting facility coolant flow through the air-to-liquidheat exchanger dependent on sensed data center air temperatureascertained by the second temperature sensor.
 3. The method of claim 1,wherein the first heat exchange assembly further comprises a coolantpump in fluid communication with the system coolant loop for pumpingsystem coolant therethrough, and wherein the method further comprisesemploying a single motor for simultaneously driving the coolant pump ofthe first heat exchange assembly and the air-moving device of the secondheat exchange assembly.
 4. The method of claim 1, wherein the hybrid airand liquid coolant conditioning unit comprises a vapor-compression heatexchange assembly, and wherein the facility coolant comprises arefrigerant, and the liquid-to-liquid heat exchanger of the first heatexchange assembly functions as a system coolant conditioning evaporator,and the air-to-liquid heat exchanger of the second heat exchangeassembly functions as an air-conditioning evaporator, and wherein thefirst heat exchange assembly further comprises a first expansion valvedisposed within a first facility coolant supply inlet line of thefacility coolant loop, and the second heat exchange assembly furthercomprises a second expansion valve disposed within a second facilitycoolant supply inlet line of the facility coolant loop, and wherein themethod further comprises separately controlling refrigerant passingthrough the liquid-to-liquid heat exchanger and the air-to-liquid heatexchanger from the facility coolant loop employing the at least onecontrol valve, and wherein the vapor-compression heat exchange assemblyfurther comprises a vapor-compression unit comprising a compressor and acondenser, the vapor-compression unit being in fluid communication withthe first expansion valve, the liquid-to-liquid heat exchanger, thesecond expansion valve and the air-to-liquid heat exchanger via thefacility coolant loop for exhausting heat from refrigerant circulatingtherethrough.